Published on

AWS VPC with custom DHCP option sets

Introduction

In my current job, I manage customer environments hosted on AWS.

A few weeks ago, one of our customers had a request that sounded simple at first. I remember reading the ticket on a Monday morning, coffee in hand, thinking "yeah, this should be quick."

They were running an internal SMTP server inside their AWS account, and its DNS records were stored in their own Route53 private hosted zone.

They wanted the applications we manage to be able to resolve those Route53 records so the applications could connect to the SMTP server.

My first thought was: "Easy. Just associate their Route53 hosted zone with our VPC." I'd done this before. Should take maybe 30 minutes, tops.

The straightforward solution? Use Route53 VPC associations.

+-------------------+
|    Application    |
+-------------------+
          |
          v
+-------------------+
|   DNS Lookup      |
+-------------------+
          |
          v
+---------------------------+
| Route53 (Customer Records)|
|  + SMTP Server Hostname   |
+---------------------------+
          |
          v
+-------------------+
|   SMTP Server     |
+-------------------+

But here's the catch:

the customer's environment we managed was using a custom DHCP options set. That meant every DNS lookup request from the applications was forwarded to our company's internal DNS server — not AWS's Route53 resolver.

I didn't realize this at first. Spent about an hour setting up the VPC association, testing it, and watching it fail. The application couldn't resolve the SMTP hostname. I checked the Route53 configuration three times. Everything looked correct.

The troubleshooting journey

This is where it got frustrating.

I SSH'd into one of the application servers and tried a simple nslookup for the SMTP server hostname. Nothing. The DNS query just timed out.

Then I tried looking up a public domain like google.com. That worked fine.

So DNS was working in general, but not for the customer's Route53 records. That's when I started digging into the VPC configuration.

Found the DHCP options set. Saw that it was pointing to our company's internal DNS servers instead of AWS's default. I remember thinking, "Oh. That's why."

The VPC association was working fine—Route53 was ready to answer queries. But the queries were never reaching Route53 because they were all going to our internal DNS server instead.

I pinged one of my colleagues who'd been at the company longer. "Hey, have you dealt with custom DHCP options before? I'm trying to get Route53 resolution working but everything's going through our internal DNS."

He said, "Yeah, that's a pain. You might need to set up conditional forwarding on the DNS server side."

That made sense, but I wanted to verify something first. I needed to know if there was a way to query AWS's DNS resolver directly, even with the custom DHCP options in place.

AWS Reserved DNS Servers

That's when I found the documentation about AWS's reserved DNS IPs.

We learned that upon creating a VPC, AWS automatically provides a built-in DNS resolver at special reserved IP addresses:

  1. 169.254.169.253 (IPv4 universal)

  2. fd00:ec2::253 (IPv6)

  3. VPC CIDR base + 2 (for example, in 10.0.0.0/16, the DNS IP is 10.0.0.2)

These resolvers work regardless of the DHCP options set.

I'd been working with AWS for over a year and had no idea these existed. The documentation mentioned them, but I'd never had a reason to use them before.

The key insight: even though the DHCP options were pointing to our internal DNS, I could still query AWS's DNS directly by using these reserved IPs.

Testing It Out

I wanted to confirm this would actually work before proposing it as a solution.

Setup:

  • VPC CIDR block: 10.0.0.0/16
  • AWS reserved DNS IP: 10.0.0.2

1. DNS lookup using our internal company DNS server

First, I tried the default DNS lookup, which was going through our internal DNS:

Error

Result: Works for internal company domains, but cannot resolve customer's Route53 records.

This confirmed what I already knew. Our internal DNS had no idea about the customer's Route53 zone.

2. DNS lookup using AWS reserved DNS server (10.0.0.2)

Then I tried querying AWS's reserved DNS IP directly:

nslookup smtp.customer-domain.internal 10.0.0.2
Success

Result: Works as expected. Able to resolve customer Route53 records.

That moment when it worked was pretty satisfying. I remember thinking, "Okay, so we can definitely reach Route53. Now we just need to make sure the right queries get there."

Finding the solution

At this point, I had a few options:

  1. Change the DHCP options to use AWS's DNS instead of our internal DNS

    • Problem: This would break resolution for our internal company domains

  2. Modify the application to use AWS's DNS for specific lookups

    • Problem: We manage multiple applications, and this would require changes to each one

  3. Set up conditional forwarding on our internal DNS server

    • Forward queries for the customer's domain to AWS's DNS
    • Keep everything else going through our internal DNS
    • No application changes needed

Option 3 made the most sense. I talked it over with our network team, and they agreed to set up the conditional forwarder.

The configuration was straightforward: any DNS query for *.customer-domain.internal gets forwarded to 10.0.0.2 (AWS's DNS). Everything else continues to resolve normally.

Resolution

Once we learned about this behavior, we decided to use the following approach:

for any DNS query targeted at the SMTP server, we forward the resolution from our internal DNS server to AWS DNS.

+-------------------+
|    Application    |
+-------------------+
          |
          v
+-------------------+
| Internal DNS      |
| (Company DNS)     |
+-------------------+
     |        |
     |        |
     |        +----------------------+
     |                               |
     v                               v
+-------------------+       +---------------------------+
| Normal Resolution |       | Conditional Forwarder to |
| (company domains, |       | AWS VPC DNS              |
| internet, etc.)   |       | (10.0.0.2 / 169.254.169.253) |
+-------------------+       +---------------------------+
                                    |
                                    v
                         +-------------------+
                         | Route53 / AWS DNS |
                         +-------------------+

The network team had the forwarder set up within a day. I tested it from the application server, and everything worked. The SMTP hostname resolved correctly, and the application could connect to the SMTP server.

Closed the ticket later that week. What I thought would take 30 minutes ended up taking about three days of troubleshooting and coordination, but we got there.

What I learned

DHCP options are more important than I thought

I used to think of DHCP options as something you set once and forget about. This issue taught me that they can have significant implications for how your applications interact with AWS services.

If you're using custom DHCP options, you need to be aware that you're opting out of AWS's default DNS behavior. That's not necessarily bad, but it means you need to handle DNS resolution more carefully.

AWS's reserved DNS IPs are a lifesaver

The fact that you can always reach AWS's DNS resolver, even with custom DHCP options, is incredibly useful. I've used this knowledge a few times since then for troubleshooting other DNS issues.

If you're ever in a situation where you need to verify that Route53 is working correctly, you can always query 10.0.0.2 (or the equivalent for your VPC) directly.

Conditional forwarding is underrated

Before this, I'd never set up conditional forwarding. Now I see it as a really elegant solution for hybrid DNS scenarios where you need to resolve both internal and AWS-hosted domains.

It's also completely transparent to applications. They don't need to know anything about the DNS setup—they just make normal DNS queries and get the right answers.

Documentation is helpful, but experience is better

The AWS documentation mentioned the reserved DNS IPs, but I didn't really understand why they mattered until I ran into this problem. Now when I read AWS docs, I pay more attention to these kinds of details.

Gotchas to watch out for

VPC peering and DNS

If you're using VPC peering, be aware that DNS resolution across peered VPCs requires additional configuration. The reserved DNS IPs only work within their own VPC.

Route53 Resolver endpoints

For more complex scenarios, you might need Route53 Resolver endpoints (inbound and outbound). These are useful if you need to resolve DNS queries between on-premises networks and AWS, or between multiple VPCs.

We didn't need them for this particular issue, but I've since learned they're the right tool for more advanced hybrid DNS setups.

Testing is crucial

Don't assume DNS is working just because the configuration looks right. Always test from the actual application server, not from your local machine or a bastion host. DNS behavior can vary depending on where you're querying from.

Final Thoughts

Oftentimes, DHCP settings are taken for granted.

This problem taught me another lesson: AWS provides hidden but powerful capabilities that can still support DNS resolution even when we step outside the default setup.

It also reminded me that "simple" requests often aren't as simple as they seem. The actual solution was straightforward once I understood the problem, but getting to that understanding took time and troubleshooting.

Now when I see a DNS-related ticket, I check the DHCP options first. Saves a lot of time.

If you're managing AWS environments with custom DHCP options, keep those reserved DNS IPs in mind. They might save you a few hours of troubleshooting someday.